Optimality Criteria in Reinforcement Learning

Sridhar Mahadevan Department of Computer Science and Engineering University of South Florida Tampa, Florida 33620 [email protected] Abstract Embedded autonomous agents, such as robots or softbots, are faced with solving sequential decision problems. Reinforcement learning (RL) is a particular approach to sequential decision problems, in which the agent-environment interaction is modeled as a Markov decision process (MDP). RL agents solve sequential decision problems by learning optimal policies for choosing actions. Thus, at the core of RL is the definition of what it means for a policy to be "optimal". In this paper, we discuss a variety of optimality metrics from the dynamic programming literature, and examine their suitability for RL. We also discuss the challenges in devising RL algorithms for the various metrics.Sequential Decision Tasks In recent years, a unifying viewpoint based on embedded autonomous agents has shaped much work in arti cial intelligence (Kaelbling 1993; Rusell & Norvig 1994). Examples of such agents range from robots operating in unstructured environments, softbots navigating the Internet, or even industrial controllers operating some complex machinery. Such agents are faced with a di cult sequential decision problem, as illustrated in Figure 1. At every time instant, the agent observes the world to be in some state (perhaps, imperfectly), and must choose an action that (stochastically) alters the state of the environment. The goal of the agent is to behave optimally, that is, it needs to choose actions using a policy (either learned or programmed by its designer). The environment typically supplies the agent with some feedback regarding its performance, such as a reward or cost function. In this paper, we will be primarily interested in in nitehorizon decision tasks, where the agent continues to operate forever. The special case of tasks that have Paper to be presented at the AAAI Fall Symposium on Learning Complex Behaviors in Adaptive Intelligent Systems, Nov. 9 11th, MIT, Boston, 1996. Autonomous Agent WORLD